The present invention generally relates to vehicle piloting; and more particularly, to the visual characteristics and behavior of a driver which is then analyzed to facilitate the driving experience and driver performance.
There is significant ongoing research related to driver fatigue, distraction, workload and other driver-state related factors creating potentially dangerous driving situations. This is not surprising considering that approximately ninety-five percent of all traffic incidents are due to driver error, of which, driver inattention is the most common causative factor. Numerous studies have established the relationship between eye movements and higher cognitive processes. These studies generally argue that eye movements reflect, to some degree, the cognitive state of the driver. In several studies, eye movements are used as a direct measure of a driver's cognitive attention level, and alternatively mental workload.
Knowing where a driver is looking is generally accepted as an important input factor for systems designed to avoid vehicular incidents, an in particularly, crashes. By ascertaining where a driver is looking, Human Machine Interaction (HMI) systems can be optimized, and active safety functions, such as forward collision warnings (FWC), can be adapted on the basis of driver-eye orientation and movements. This may be done as an offline analysis of many subjects, or using an online, or real-time algorithm to perhaps adapt such things as FCW thresholds to the current driver state.
Drivers of all types of vehicles are often unaware of the effects that drowsiness and distraction have on their own abilities for vehicle control. Humans in general, and particularly as drivers, are poor judges of their own performance capabilities. Typically, a driver's self-impression of his or her capabilities is better than actuality. Even persons who have basically good driving skills, will not perform uniformly at all times when behind the wheel of a vehicle. Furthermore, there are many times during driving trips that very little demand is placed on the driver with respect to execution of driving tasks. As a result, drivers are lulled into states of mind where little attention is being devoted to the driving task. Not surprisingly, driver inattention is a leading cause of vehicular collisions, and especially automotive collisions. According to a Nation Highway and Transportation Safety Administration (NHTSA) study of over two and one-half million tow-away crashes in a year's time, driver inattention is a primary cause of collisions that accounts for an estimated twenty-five to fifty-six percent of crashes. In that study, inattention was defined as having three components: visual distraction, mental distraction (looking without seeing) and drowsiness. Common crash types caused by inattention are: rear-end collisions, collisions at intersection, collisions while lane changing or merging, road departures, single vehicle crashes, and crashes that occur on low speed limit roadways.
Drowsy drivers are a well known phenomenon. At least one survey has indicated that fifty-seven percent of drivers polled had driven while drowsy in the previous year, and twenty-three percent had actually fallen asleep at the wheel. It is known that fatigue impairs driver performance, alertness and judgment. Collisions caused by drowsiness are a serious road safety problem, and fatigue has been estimated to be involved in as many as twenty-three percent of all crashes.
From a technological perspective, there is an ongoing and rapid increase of new information systems and functionalities that may be used within vehicles including mobile telephones, navigation aids, the internet, and other types of electronic services. The effect of mobile telephone use on drivers has been foremost in the public eye because of their widespread use, but sales of navigation aids and IT services are also growing fast. Mobile telephones alone have been estimated to have caused 300-1000 fatalities in one years time in the United States, and this is projected to reach 4000 fatalities per year in 2004. Distractions such as handheld telephone use, sign reading, eating food, interaction with other passengers, observing objects and manipulating devices in-the vehicle have the potential for capturing a driver's attention in an excessive way and thus also compromising safety. It is especially important that driving safety not be compromised as these new types of services and activities become more common place in the driving environment.
Driver workload increases based on utilization of these new functionalities and technologies. In this context, “workload” should be understood to refer to how busy a person is and the amount of effort they need to perform required tasks. When a driver has many things to do and is experiencing high workload, a high attention demand is being made on the driver in that there is much to be done at the same time. Drivers often attend to things that are not related to driver control of the vehicle and are therefore technically irrelevant to the driving situation. These things are often called secondary tasks and are potential distracters from driver attention to primary driving tasks. A secondary task becomes a distraction (including visual-, auditory-, cognitive-, and biomechanical distractions) when the driver's attention is captured thereby to a degree that insufficient attention is left for the primary control tasks of driving. As a result, driving performance such as lane keeping and speed control are compromised as ultimately is safety.
Driving tasks and secondary tasks overlap in the sense that some secondary tasks are driving related as diagrammatically shown in FIG. 1. Two difficulties arise from this relationship between the driving and secondary tasks. First, it can be difficult to delineate which secondary task information is “irrelevant to the driving situation” and which is not; and second, certain driving related secondary tasks, for instance, looking for a street sign or planning a driving route may also compromise safety as graphically depicted in FIG. 1.
It should also be appreciated that the driver is often unaware of the effects of distraction on the driving task. Also, drivers cannot reliably determine when they are impaired by fatigue to the point of having a serious vigilance lapse or uncontrolled sleep attacks. The attention management systems outlined herein are intended to increase safety by assisting the driver in drowsy, distractive, and/or high workload situations.
As mentioned above, an interesting use for eye movements is in the ergonomics and HMI fields. For instance, such utilization may be made in determining best placements for Road and Traffic Information (RTI) displays, as well as analyzing whether a certain HMI poses less visual demand than another. These types of analysis can, and are made by studying subjects eye movements while using the device HMI. A primary drawback associated with current methods, however, is that there are few, if any, suitable automated tools for performing the analysis; in their absence, resort is commonly made to labor intensive, manual analysis.
A significant problem in current eye movement research is that every research team seems to use their own definitions and software to decode the eye movement signals. This makes research results very difficult to compare between one another. It is desirable to have a standard that defines visual measures and conceptions. ISO 15007 and SAEJ-2396 constitute examples of such standards in that they prescribe in-vehicle visual demand measurement methods and provide quantification rules for such ocular characteristics as glance frequency, glance time, time off road-scene-ahead and glance duration, and the procedures to obtain them. However, the two standards are based on a recorded-video technique, and rely on frame-by-frame human-rater analysis that is both time consuming and significantly unreliable. As the number of various in-vehicle information and driver assistance systems and devices increases, so will the probable interest for driver eye movements and other cognitive indicators. Thus, the need for a standardized, automated and robust analysis method for eye movements exists, and will become even more important in the future.
Certain eye tracking methods and analysis procedures have been statistically verified to the prescriptions of ISO 15007 and SAEJ-2396. These physical portions of the systems can be configured to be neither intrusive nor very environmentally dependent. At least one example is based on two cameras (a stereo head) being positioned in front of the driver. Software is used to compute gaze vectors and other interesting measures on a real-time basis, indicating such things as head position and rotation (orientation), blinking, blink frequency, and degree of eye-openness. Among other important features in this software are the real-time simultaneous computation of head position/rotation (orientation) and gaze rotation; a feature that has never before been available. Also, it is not sensitive to noisy environments such as occur inside a vehicle. Among other things, “noise” in the data has been found to be a significant factor impacting data-quality-degradation due to such things as variable lighting conditions and head/gaze motion.
It may seem that the previous work done in the area of eye tracking related research is reasonably exhaustive. Yet, as progress is made enabling eye tracking to be more robust and portable, this technology area continues to expand. There are, however, not many on-road studies of driving task-related driver characteristics, and to date, there has been no utilization of eye-tracking data on a real-time basis to calculate measures such as visual or cognitive distraction (see FIGS. 2-4). This is at least partially the result of the time consuming nature of manual segmentation and/or technical difficulties related to the non-portability of commonly used eye-tracking systems. However, in studies conducted in laboratory environments, a variety of algorithms have been developed. Many different approaches have been taken using, for example, Neural Networks, adaptive digital filters, Hidden Markov Models, Least Mean Square methods, dispersion or velocity based methods and other higher derivative methods. Many of these methods, however, are based on the typical characteristics of the eye tracker, such as sampling frequency, and do not work well with other such systems.
Heretofore, there has been no standard for defining what driver characteristic(s) are to be measured, and how they are to be measured. There is no standard that refers to the basic ocular segmentations including saccades, fixations, and eye closures. The standard only concerns glances; that is, the incidence of rapid eye movement across the field of vision.
Interestingly, no current methods take into account smooth eye movements or pursuits; that is, purposeful looks away from the driving path such as looking (reading) a road sign as it is passed. In fact, many studies are designed so that smooth pursuits will never occur, such as by assuring that there are no objects to pursue. This avoidance by current research is understandable; it can be difficult to differentiate a smooth pursuit from a saccade or a fixation. These characteristics are rarely mentioned in the literature. Regardless of the reason(s) that these characteristics have not been considered, smooth pursuits are taken into account with regard to the presently disclosed invention(s) because such smooth eye movement does occur quite often under real driving conditions.
Fundamental to driving a vehicle is the necessity to aim the vehicle, to detect its path or heading, and to detect potential collision threats whether they are from objects or events. This road scene awareness is a prerequisite to longitudinal and lateral control of the vehicle. It should be appreciated that road-center is not always straight ahead of the longitudinal axis of the vehicle, but is often off-centerline due to curves that almost always exist in road-ways to greater and lesser degrees. Even so, research shows that drivers tend to look substantially straight ahead (considering reasonable deviations for road-curvature), with their eyes on the road most of the time; that is, about eight-five to ninety-five percent of the time. Still further, prudence tells the average driver that glances away from the road center or travel path are best timed not to interfere with aiming the vehicle, and to coincide with a low probability of an occurrence of unexpected event or object encounter. Even so, the statistics above demonstrate that even prudent drivers are not always attentive to driving demands, nor are they consistently good managers of their own work loads and distractions when driving.
The theoretical basis for the road center concept considers that the visual guidance of vehicle control is based on optical flow information in the forward roadway region. In order to receive the most relevant visual information, drivers tend to fixate on specific locations, or “anchor points”. It has been proposed that information is mainly obtained from two such anchor points: one far point and one near point (e.g. Salvucci and Gray, 2004). For the far region, it has been suggested that the most efficient anchor point is the target that steering is directed to (Wilkie and Warm, 2005), although other anchor points are possible as well (see Victor, 2005 for a review of the literature). The near point is located in the region just ahead of the vehicle (Salvucci and Gray, 2004). The far point has been proposed to account for the rotational component of the optical flow, while the near point is better suited for uptake of the translational component (Victor, 2005).
The region defined by the anchor points is here conceptualized as the road-center (RC). During normal driving, the driver usually shares the visual attention between the road center and other sources of information, e.g. the mirrors, road signs, or other objects inside and outside the vehicle. However, during extended visual time sharing, e.g. when performing a task on an in-vehicle information system (IVIS), the on-road glances need to be focused on the regions most relevant for path control, i.e. the anchor points. This results in a strong concentration of the road-ahead glances (Victor et al., 2005). As mentioned above, this is one of the key motivations for using road-center glances as the basis for visual demand measurement. The second key motivation, also confirmed by empirical results (Victor et al., 2005), is that the great majority of off-RC glances during IVIS task performance are towards the IVIS target.
Road Center Identification—It is important to note that the location of the road center, from the driver's point of view, is determined by the position/orientation of the body and the vehicle relative to the environment. Thus, a substantial amount of variation of the road center is induced by differing physical dimensions of a driver, seating postures, as well as road curvature. For this reason, the RC is estimated from the data in a bottom-up fashion.
Driving is not a particularly demanding task in most instances. For example, it is estimated that during most interstate driving, less than fifty percent of a driver's perceptual capacity is used. Because of this, drivers often perform secondary tasks such as dialing cellular phones and changing radio channels. When secondary tasks are performed, a timesharing glance behavior is exhibited in which the eyes are shifted back and forth between the road and the task. This temporal sharing of vision is an implication of having a single visual resource. One could say that the road is sampled while performing secondary tasks instead of the opposite. The problem, which induces collisions, is that unexpected things might happen during the interval when the eyes are off the road and reactions to these unexpected events or objects can be seriously slowed.
The new measures and analysis techniques presented herein exploit this fundamental and necessary driving eye-movement behavior of looking straight ahead or on the vehicle path trajectory. The measures give an accurate off-line assessment of the visual impact of performing visually, cognitively, or manually demanding in-vehicle tasks that have been found to be highly correlated with conventional measures. They also enable a comparison with normal driving. The measures presented herein are importantly also suitable for on-line calculation and assessment of this visual impact and thus represent real-time measures that can be used for distraction and work-load detection. US 2005/0073136 A1 discloses a method for analyzing ocular and/or head orientation characteristics of a subject. A detection and quantification of the position of a driver's head and/or eye movements are made relative to the environment. Tests of the data are made, and from the data locations of experienced areas/objects of-subject-interest are deduced. By utilizing gaze direction data, regardless of whether it is based on head orientation or eye (ocular) orientation, the relative location of the road center and the instrument cluster can be deduced for a particular driver. A concept of identifying the road center is disclosed. WO 03/070093 A1 discloses a system and a method for monitoring the physiological behaviour of a driver that includes measuring a physiological variable of the driver, assessing a drivers behavioural parameter on the basis of at least said measured physiological variable and informing the driver of the assessed driver's behavioural parameter. The measurement of the physiological variable can include measuring a driver's eye movement, measuring a driver's eye-gaze direction, measuring a driver's eye-closure amount, measuring a driver's blinking movement, measuring a driver's head movement, measuring a driver's head position, measuring a driver's head orientation, measuring a driver's movable facial features, and measuring a driver's facial temperature image.
At least one characteristic of the present intention(s) is the provision of validated analysis methods and algorithms that facilitate: automated analysis of behavioral movement data produced by head/eye/body-tracking systems, substantial elimination of human rating, and outputting filtered and validated characteristic data that is robust against errors and noise. Preferably, these facilitations are conducted in accordance with ISO/SAE and similarly accepted present and future standards. Certain algorithms, standards, and facilitations are discussed in U.S. application Ser. No. 10/605,637, filed Oct. 15, 2003 the contents of which are herein incorporated by reference in its entirety.
The present invention provides for a method on analyzing data that is sensed based on the physiological orientation of a driver in a vehicle. The data is descriptive of the driver's gaze-direction and can be defined by a data set. The data is processing using a computer, and from at least a portion of that data, criteria defining a location of driver interest is determined. Based on the determined criteria, gaze-direction instances are classified as either on-location or off-location. The classified instances can then be used for further analysis related to the location of visual interest. Further analysis generally relates to times of elevated driver workload and not driver drowsiness.
A location can be any location of interest, for example a location may include: the road center, a location behind the driver, a location to the left or right of the driver, a rear view mirror, a side mirror, a center console, a car accessory (e.g. radio, window switch, navigation system), a personal accessory (e.g. cell phone, PDA, laptop), or a passenger (e.g. children in car seats or back seat). The above list is not all-inclusive and is provided to show just a few examples of location. As seen from the examples above, the location need not be fixed, but can change with time, for example when the location is a cellular telephone or PDA the location changes with time when the user dials the phone, answers the phone, checks caller ID, checks incoming messages, or sends outgoing messages.
The classified instances are transformed into one of two binary values (e.g., 1 and 0) representative of whether the respective classified instance is on or off location. The uses of a binary value makes processing and analysis more efficient.
Furthermore, the present invention allows for the classification of at least some of the off-location gaze direction instances to be inferred from the failure to meet the determined criteria for being classified as an on-location driver gaze direction instance.
The present invention provides for gaze-direction instances can be sensed and derived from behavioral movements. For example, the gaze-direction instances can be derived from a sensed orientation of: an above-waist portion of the driver's body; an upper torso portion of the driver's body; the head of driver; and/or at least one eye of the driver. Sensors for measuring behavioral movements include a variety of sensors, including, inter alia, cameras, ultrasonic sensing devices, and capacitive sensors.
As seen above, an aim of the present invention is to provide simplified characterization rules which characterize data as either on or off a specified location. In one exemplary embodiment, the characterization is either a road-center visual fixation, or a non-road-center visual fixation. A road-center visual fixation is generally characterized when the driver is looking forward in a typical driving fashion, i.e. the driver is visually fixated on the road-center. Non-road-center visual fixations, where the driver is looking away from the road-center, can be inferred from visual fixations that are not characterized as road-center visual fixations.
In another exemplary embodiment, the characterization is either a rear-view-mirror visual fixation, or a non-rear-view mirror visual fixation. A rear-view-mirror visual fixation is generally characterized when the driver is looking in to the rear view mirror to look behind the vehicle. Non-rear-view-mirror visual fixations, where the driver is not looking in to the rear-view-mirror, can be inferred from visual fixations that are not characterized as rear-view-mirror visual fixations.
Another aim is to adapt certain algorithms to a real-time environment. Another is to identify and provide driver supports that are based on visual behavior and that can assist the driver avoid potentially detrimental situations because of implemented systems that refocus the driver.
In one aspect, the present invention addresses the need for having one standard reference in a vehicle from which various objects and areas that might be of interest to a driver can be located relatively located. A standard frame of reference (defined by relative position/location/orientation {in the context of the present disclosure, utilization of the forward slash mark, /, is utilized to indicate an “and/or” relationship} within the vehicle's interior) to which head/facial/eye tracking data taken from operators of varying size, stature and behavior can be translated is desirable in that it “standardizes” such data for elegant processing for the several purposes described herein.
In at least one embodiment, the presently disclosed invention may be defined as a method for analyzing ocular and/or head orientation characteristics of a driver of a vehicle. It should be appreciated that the analysis techniques or processes described are contemplated as being capable of being applied to stored tracking data that has typically been marked with respect to time, or real-time data, which by its nature, considers time as a defining factor in a data stream; hence the descriptive name, “real-time” data. In any event, this embodiment of the invention contemplates a detection and quantification of the position of a driver's head relative to the space within a passenger compartment of a vehicle. A reference-base position of a “benchmark” driver's head (or portion thereof) is provided which enables a cross-referencing of locations of areas/objects-of-driver-interest relative thereto. It should be appreciated that these areas/objects-of-driver-interest may be inside or outside the vehicle, and may be constituted by (1) “things” such as audio controls, speedometers and other gauges, and (2) areas or positions such as “road ahead” and lane-change clearance space in adjacent lanes, in order to “standardize” the tracking data with respect to the vehicle of interest, the quantification of the position of the driver's head is normalized to the reference-base position thereby enabling deducement of location(s) where the driver has shown an interest based on sensed information regarding either, or both of (1) driver ocular orientation or (2) driver head orientation.
In another embodiment, the presently disclosed invention presents the general concept of road-center (RC) based measures, where visual demand is quantified in terms of glances away from the road center, for both off-line and on-line (real-time) applications. The main advantage of this simplification is that one can allow for lower data quality during glances away from the road (since gaze outside of the RC area, is ignored).
In the event that tracking information is available on both driver head and eye characteristics, sensed information regarding driver ocular orientation is preferentially utilized as basis for the deducement of location(s) of driver interest. A switch is made to sensed information regarding driver head orientation as basis for deducing where driver interest has been shown when the quality of the sensed information regarding driver ocular orientation degrades beyond a prescribed threshold gaze confidence level. As an example, this switch may be necessitated when the driver's eyes are occluded; that is, obscured or covered in some way that prevents their being tracked. The condition of being occluded is also contemplated to include situations in which the tracking sensor(s) is unable to track the eyes because, for example, of an inability to identify/locate relative facial features. For example, eyes-to-nose-to-mouth orientation and reference cannot be deduced (some tracking systems require that a frame of reference for the face be established in order to locate the eyes which are to be tracked and characterized by data values. When the face is not properly referenced, it is possible for some sensor systems to track, for instance, the subject's nostrils, which have been confused for the eyes, or eye-glasses that are being worn distort (refractionally) or obscure (sunglasses) the eye-image. Another example of the eyes being occluded is when the driver's head position departs away from an eyes-forward (predominant driving) orientation beyond an allowed degree of deviation. In these events, the eye(s) of the driver are effectively visually blocked from the tracking equipment (sensors) that is generating the eye-orientation data.
Preferably, a mathematic transformation is utilized to accomplish the normalization of the quantification of the position of the driver's head to the reference-base position. In an onboard installation, it is preferred that the mathematic transformation be performed using a vehicle-based computer on a substantially real time basis.
Probable positions of areas/objects-of-driver-interest relative to the reference-base position are prescribing, in this regard, such prescriptions act as templates against, or onto which the sensed data can be read or overlaid.
Alternatively, probable positions of areas/objects-of-driver-interest are defined relative to the reference-base position based on sensed driver ocular characteristics. In one exemplary development, such definitions of probable positions of areas/objects-of-driver-interest relative to the reference-base position can be established based on the sensed driver ocular characteristic of gaze frequency. Here, establishment of the gaze frequency is based on quantification of collected gaze density characteristics.
In one embodiment of the invention, an area/object-of-driver-interest (which is intended to be interpreted as also encompassing a plurality of areas/objects-of-driver-interest) is identified based on driver ocular characteristics (exemplarily represented as tracking data) by mapping the sensed driver ocular characteristics to the prescribed or defined probable locations of areas/objects-of-driver-interest relative to the reference-base position. That is, identification of an object or area that has been deduced as probably being of interest to a driver can be made by comparison of the observed data (head and/or eye tracking data) to a prescribed template as defined hereinabove, or by comparison to a known data set that has been correlated to particular objects and/or areas in which a driver would be potentially interested:
One example would be that an area-based template devised for a particular vehicle, and relative frequencies at which a driver looks at various locations/object is identified. For instance, it may be found that a typical driver looks in a substantially straight-forward direction about forty percent of driving time and the gauge cluster, including the speedometer about twenty percent of driving time. It is also known that spatially, the center of these two areas is one below the other. Therefore, utilizing gaze direction data (regardless of whether it is based on head orientation or eye (ocular) orientation), the relative location of the road center and the instrument cluster can be deduced for a particular driver. Once that basic frame of reference is established, correspondence to reality for the particular vehicle can be deduced, and a translation to a reference frame can be determined. Still further, glances to the vehicle's audio controls can also be deduced, for instance, if statistically, it is known that a typical driver looks to the audio controls approximately ten percent of normal driving time. Once a period of “learning time” has been recorded, the relative locations of many areas/objects-of-driver-interest can be ascertained on a statistical basis; even independent of any known map of objects/areas, or reference frame in the vehicle.
In another aspect, the disclosure describes tailoring prescribed functionalities performed by the vehicle based on the mapped driver ocular characteristics. This may be as simple as adapting a distraction warning to sound when it is detected that the driver has looked away from the road too long, to causing an increase of the buffer zone maintained behind a leading vehicle by an adaptive cruise control system.
It has been discovered that these areas/objects-of-driver-interest can be identified based either in part, or exclusively on sensed information regarding driver ocular orientation exclusively constituted by a measure of gaze angularity. With respect to at least a reference frame within a particular vehicle (exemplarily identified as a particular make and model of an automobile), angular location of an area/object is particularly elegant because the need to consider distances are removed. That is to say, if an area-location were to be identified as statistically (probabilistically) representing an area/object of probable driver interest, the distance at which that area is located away from the reference frame must be known. This turns on the fact that a defined area expands from a focal point much like a cone does from its apex. An angle from the apex, however, is a discrete measure (see FIG. 5).
The measure of gaze angularity can be derived from a sensed eyeball-orientation-based gaze-direction vector. This could be taken from the observation of one eyeball, but preferably, it is taken as a conglomeration of observations taken from both eyeballs. Therefore, the representative vector is more accurately described as a vector emanating from the region of the subjects nose bridge, and oriented parallel to an average of observed angularity. Furthermore, a measure of gaze angularity could be estimated from the observation of head, face, or other body movements and/or positions.
While the invention has been described with respect to particulars in terms of eyeball angularity herein above, it is also contemplated that related, if not similar results can be obtained from making similar observations based on head orientation. In general, the comparison can be described as using the direction in which the nose points (head-based), as opposed to the direction in which the eyes are oriented from the reference frame defined by the orientation of the reference frame, defining probable positions of areas/objects-of-driver-interest relative to the reference-base position based on sensed head orientation.
In at least one embodiment, the definitions of probable positions of areas/objects-of-driver-interest is determined relative to the reference-base position based on sensed head orientation from which a face-forward direction is deduced. In this case, as with eyeball trajectory measurement data, particular head orientations, and hence a face-forward direction can be established utilizing density mappings indicative of frequency at which a driver looks in a certain direction.
Objects/areas-of-driver-interest can be identified by correlating the representative mapping (therefore, this can also be accomplished from the direct data of angularity) against prescribed/defined probable locations of areas/objects-of-driver-interest relative to the reference-base position.
When addressing head orientation-based analysis, the measure of gaze angularity can be derived from a sensed head-orientation-based gaze-direction vector.
In another embodiment, the invention takes the form of a method for developing a bench-mark (reference frame) for comparison in assessing driver activity and/or driver condition. This method comprises (includes, but is not limited to) collecting (which may also include using a stream of recorded data) a stream of gaze-direction data based on a sensed characteristic of a driver, and based on density patterns developed therefrom, defining gaze-direction-based parameters corresponding to at least one region of probable driver interest.
As before, this method entails utilizing measures of at least one of (1) driver ocular orientation and (2) driver head orientation to constitute the gaze-direction data.
A region representative of typical eyes-forward driving is established based on a high-density pattern assessed from the collected gaze-direction data. Exemplarily, the region may be defined as an area defined in two dimensions such as a parabola or a volume defined in three dimensions such as a cone radiating from the reference frame with an apex thereof essentially located at eye-position of a typified driver relative to an established reference frame.
The collected gaze-direction data is compared to the established representative region, and thereby identifying gaze departures based on the comparison. Based on similar comparison, other qualities of the environment or the driver may be deduced. For example, the gaze-direction data can be used to identify and/or measure such things as driver cognitive distraction, driver visual distraction, and/or high driver work load conditions.
Still further, the method contemplates and provides means for quantifying the severity (degree) of a driver's impairment with respect to performing driving tasks based upon an ascertained frequency or duration (depending on whether occurrences are discrete or continuous incidents) at which such an indicative condition as gaze departure, cognitive distraction, (3) visual distraction and (4) high driver work load is detected in a prescribed time period.
The incidents of interest can be logged, stored and/or transmitted for further analysis by a processor. Conversely, the data representative of the incidents of interest can be analyzed on a real-time basis either locally, or remotely if also transmitted in real-time.
Attention management systems and methods have as an objective to increase safety by assisting drivers in drowsy, distractive, and/or high workload situations. Functional specifications are provided for a number of attention management systems that can be characterized to include drowsiness managers, distraction managers, managers for distraction adaptation of forward collision and lane change warning systems, and workload managers that are at least in part controlled based on driving demand estimations observed or deduced from visual behavior of the driver. A hardware system that can be suitably employed to perform these driver attention management tasks is also described. A “platform” for development of the instant drowsiness and distraction manager based on Human Machine Interaction (HMI) is also disclosed, as is description of continuous and post-trip attention feedback systems. The HMI approach has as an objective thereof to counteract driver inattention by providing both imminent collision warnings, as well as attention-feedback to cause positive behavioral change.
At least one utilization of such analysis is to provide driver feedback when the severity quantification exceeds a prescribed severity threshold level. For instance, a driver may be warned when excessive levels of visual distraction (too much looking away) or cognitive distraction (not enough looking away—staring ahead when preoccupied) occur.
Another utilization of the output from the analysis is to tailor prescribed functionalities performed by the vehicle when the severity quantification exceeds a prescribed severity threshold level. An example would be causing an adaptive cruise control system to institute additional space between a leading vehicle when the driver is assessed to be distracted or inattentive.
One particularly advantageous mode for analyzing the stream of collected gaze-direction data is the utilization of a primary moving time-window of prescribed period traversed across the data series (a well known analysis tools to those persons skilled in the statistical analysis arts), and detecting characteristics within the primary moving time-window indicative of an occurrence of driver time-sharing activity. An example is taking an average of certain data within a moving ninety second window. As the window progresses along the data series, new data is added to the consideration and the oldest data is disregarded (new-in and old-out in equal amounts, based on time).
Utilization of this process can be used to identify periods of high driver workload based on a frequency of threshold-exceeding occurrences of driver time-sharing activity. In order to rid the window of the effect of the detected occurrence, refreshment (flushing or restoring to normal) of the primary moving time-window upon the detection of cessation of an occurrence of driver time-sharing activity is caused. In this way the effect of the occurrence is minimized after detection and analysis, thereby readying the system for a next departure from normal.
As will be discussed in greater detail hereinbelow, several characteristics of ocular activity can be identified based on observed eye activity. Some common characteristics easily recognized by the lay person are blinking and glances. What may not be as readily appreciated by the lay person is that such things as a glance may be characterized or identified based upon lesser known constituent eye-activities such as saccades, fixations and transitions, each of which have measurable defining characteristics.
In another embodiment, the invention takes the form of a method for automated analysis of eye movement data that includes processing data descriptive of eye movements observed in a subject using a computer-based processor by applying classification rules to the data and thereby identifying at least visual fixations experienced by the subject. These rules or characteristics are discussed in greater detail hereinbelow. Analysis is also made of gaze-direction information associated with the identified fixations thereby developing data representative of directions in which the subject visually fixated during the period of data collection that is presently being analyzed.
Applied classification rules comprise at least criteria defining fixations and transitions. The classification rules can also providing criteria to define saccades are additionally utilized.
The data can be segregated, based at least partially on gaze-direction of fixations, into delimited data sets, each delimited data set representing an area/object-of-subject-interest existing during the period of data collection.
In another respect, glances are identified by applying at least one glance-defining rule to the data, each of the identified glances encompassing at least one identified fixation. In this aspect of the invention, the glance-defining rule is generally defined by at least one of the following characteristic including: glance duration, glance frequency, total glance time, and total task time.
In another aspect, a relative density is assessed of one glance set in comparison to at least one other glance set, and based thereupon, the method identifies the represented area/object-of-subject-interest of the compared glance set.
In a similar regard, the inventive method contemplates assessing a relative density of at least one glance set among a plurality of glance sets, and based upon a mapping of the assessed relative density to known relative densities associated with settings of the type in which the eye movement data was collected, identifying the represented area/object-of-subject-interest of the compared glance set. For example, using the percentages for known dwell periods on certain objects or areas of driver interest during normal driving conditions, those objects or areas can be identified from the collected data.
In another aspect, relative densities of at least two glance sets developed from data descriptive of eye movements observed in a spatially known setting are assessed and the represented area/object-of-subject-interest of each of the two compared glance sets is ascertained therefrom. Locations of the represented areas/objects-of-subject-interest are then ascertained in the known setting thereby establishing a reference frame for the known setting because the deduced locations can be mapped or overlaid on known locations of the objects/areas.
In a particularly preferred embodiment, the subject is a driver of a vehicle, and based on a density of at least one of the glance data sets, an eyes-forward, normal driver eye orientation is deduced.
A further aspect of the invention in which a vehicle driver is the subject, contemplates utilizing a plurality of analysis protocols, the selection of which is dependent upon prevailing noise characteristics associated with the data set being processed.
In one development, a first data filter of predetermined stringency is applied to an input stream of data comprising the data descriptive of eye movements observed in a driver of a vehicle. The computer-based processor is utilized, and therefrom, a first filtered data stream is outputted that corresponds to the input stream of data. (This concept of correspondence can be one in which each outputted value corresponds to the inputted value from which the outputted value is derived. Quality of the outputted first filtered data stream is assessed by applying a first approval rule thereto, and data of the outputted first filtered data stream passing the first approval rule being outputted and constituting an approved first stream of data.
In a further development, a second data filter is applied to the input stream of data that is of greater stringency (more smoothing to the data) than the first data filter utilizing the computer-based processor; and therefrom, a second filtered data stream is outputted that corresponds to the first filtered data stream via its common derivation from the input stream of data (again, correspondence/comparison based on having been computed from the same input data value). Quality of the outputted second filtered data stream is assessed by applying a second approval rule thereto, and data of the outputted second filtered data stream that passes the second approval rule is outputted and constitutes an approved second stream of data.
From the two approved data streams, a collective approved stream of data is composed that is constituted by an entirety of the approved first stream of data, and the collective approved stream of data being further constituted by portions of the approved second stream of data corresponding to unapproved portions of the outputted first filtered data stream.
In at least one embodiment, the first and second approval rules are the same; in another, the first and second approval rules are based on the same criteria, but may not be the same rules.
In a further development, the method comprises selecting at least two analysis protocols to constitute the plurality from a group consisting of: (1) a velocity based, dual threshold protocol that is best suited, relative to the other members of the group, to low-noise-content eye and eyelid behavior data; (2) a distance based, dispersion spacing protocol that is best suited, relative to the other members of the group, to moderate-noise-content eye and eyelid behavior data; and (3) an ocular characteristic based, rule oriented protocol that is best suited, relative to the other members of the group, to high-noise-content eye and eyelid behavior data.
In an associated aspect, the selection of protocols for any given data set is biased toward one of the three protocols in dependence upon a detected noise level in the data set. In another aspect, the rule oriented protocol considers one or more of the following standards in a discrimination between fixations and saccades: (1) fixation duration must exceed 150 ms; (2) saccade duration must not exceed 200 ms; and saccades begin and end in two different locations.
In a further regard, quality of the data descriptive of behavior movement is assessed based on relative utilization of respective analysis protocols among the plurality of analysis protocols. Alternatively, or in association therewith, the quality assessment can be made considering time-based, relative utilization of respective analysis protocols among the plurality of analysis protocols over a prescribed time period.